Apparatus for calculating optical flow and camera motion using correlation matching and system model in moving image

ABSTRACT

Disclosed is an apparatus for calculating an optical flow and a camera motion using a correlation matching and a system model in a moving image, in which the calculation of the optical flow is executed in real time, and even though an image rotation or zoom phenomenon is happened the optical flow can be calculated by use of the correlation matching and system model. The apparatus comprises a feature point extracting section for extracting a feature point of a sequence of input images (previous image and current image); an optical flow calculating section for calculating an optical flow by use of the feature point extracted by the feature point extracting section; a first camera motion calculating section for calculating a camera motion by use of the optical flow calculated by the optical calculating section; a second camera motion calculating section for eliminating an incorrect optical flow among the optical flows calculated by the optical flow calculating section and again calculating an information of the camera motion; an optical flow location estimating section for estimating a location of the optical flow by estimating an information of a distance difference between a feature point estimated location and a current feature point location in the previous image according to the camera motion calculated by the second camera motion calculating section; and a weighted value calculating section for calculating a weighted value according to the camera motion calculated by the second camera motion calculating section and providing the weighted value to the optical flow calculating section.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The present invention relates to an apparatus for calculating anoptical flow and a camera motion using a correlation matching and asystem model in a moving image, and more particularly, to an apparatusfor calculating an optical flow and a camera motion using a correlationmatching and a system model in a moving image, in which after theoptical flow is acquired from a sequence of moving images, the cameramotion is calculated.

[0003] 2. Background of the Related Art

[0004] Up to now, in order to calculate an optical flow by extractingfeature points in a sequence of moving images, a gradient-basedapproach, a frequency-based approach, and a correlation-based approachare proposed.

[0005] Hron and Schunck algorism is typical of the gradient-basedapproach.

[0006] Such an approach is to find a pixel point in which its value isminimized according to a variation of a peripheral pixel gray value anda variation of a gray value between image frames. The greatestdisadvantage of the algorism is that since the gray value existed in acurrent frame has to be in a next frame and an object should be moved,if any, it is difficult to apply to the quick camera motion and thus tothe system in rear time.

[0007] The frequency-based approach of calculating the optical flow usesa differential value of all of pixel values in the image by employing aband-pass filter for a velocity such as a Gabor filter. This approachalso has the same disadvantage as that of the gradient-based approach.

[0008] The correlation-based approach is applied to a method ofsearching a moving object at MPEG image. The approach has many errorswhen the image is rotated or a zoom phenomenon is happened. If thephenomenon is happened, it is required an auxiliary method.

[0009] After all, according to the existing methods of calculating theoptical flow, it is impossible to calculate it in real time, because ofcalculating the optical flow for all of pixels in the image. The opticalflow calculation using the correlation matching has a drawback in thatupon occurring the image rotation or the zoom phenomenon an error ishappened on calculating the optical flow.

SUMMARY OF THE INVENTION

[0010] Accordingly, the present invention is directed to an apparatusfor calculating an optical flow and a camera motion using a correlationmatching and a system model in a moving image that substantiallyobviates one or more problems due to limitations and disadvantages ofthe related art.

[0011] An object of the present invention is to provide an apparatus forcalculating an optical flow and a camera motion using a correlationmatching and a system model in a moving image, in which the calculationof the optical flow is executed in real time, and even though an imagerotation or zoom phenomenon is happened the optical flow can becalculated by use of the correlation matching and system model.

[0012] To achieve the object and other advantages, the present inventionemploys a method of calculating the optical flow on the basis of acorrelation-based approach and using an estimated value, of which the anoptical flow calculated in a previous frame is to be positioned in acurrent frame, on the basis of the system model and the camera motioncalculated in the previous frame on occurring the image rotation or thezoom phenomenon.

[0013] In addition, since the real time process is impossible when theoptical flow of the all of pixels is calculated, the number of theoptical flows is controlled not to be increased above a certain numberby use of a SUSAN edge operation. When calculating the optical flow, themethod of combining a correlating matching value with an optical flowlocation estimated value is characterized in that a correlation matchingweighted value is reduced depending upon a rotating and moving value ofan optical axis of a CCD camera.

[0014] According to one aspect of the present invention, there isprovided an apparatus for calculating an optical flow and a cameramotion using a correlation matching and a system model in moving images,the apparatus comprising: feature point extracting means for extractinga feature point of a sequence of input images (previous image andcurrent image); optical flow calculating means for calculating anoptical flow by use of the feature point extracted by the feature pointextracting means; first camera motion calculating means for calculatinga camera motion by use of the optical flow calculated by the opticalcalculating means; second camera motion calculating means foreliminating an incorrect optical flow among the optical flows calculatedby the optical flow calculating means and again calculating aninformation of the camera motion; optical flow location estimating meansfor estimating a location of the optical flow by estimating aninformation of a distance difference between a feature point estimatedlocation and a current feature point location in the previous imageaccording to the camera motion calculated by the second camera motioncalculating means; and weighted value calculating means for calculatinga weighted value according to the camera motion calculated by the secondcamera motion calculating means and providing the weighted value to theoptical flow calculating means.

[0015] The feature point extracting means comprises a SUSAN edge drivingportion for extracting an edge image from the sequence of the inputimages; and a local max portion for selecting a largest value in a masksize region set in the edge image extracted from the SUSAN edgeextracting portion to extract a certain number of feature points. Thefeature points are extracted by the local max portion in accordancewith:$S = {\left\{ \sum_{e^{- {(\frac{{I{({x,y})}} - {I{({{x + {dx}},{y + {dy}}})}}}{T})}^{6}}} \right\}/G}$

[0016] wherein, I is a gray value, T is a threshold value for adifference between the gray values, and G is an edge strength differencebetween feature point locations of the previous and current images.

[0017] The optical flow calculating means comprises a combining sectionfor calculating a connecting strength of an edge by use of theinformation of the distance difference between the current feature pointlocation and the feature point estimated location in the previous imageprovided from the optical flow estimating means, the weighted values ofthe correlation, location incorrectness, and edge strength matchingprovided from the feature point extracting means, and the edge strengthdifference between the feature points of the previous and current imagesprovided from the feature point extracting means, and a feature pointextracting section for extracting the feature points of the largestconnecting strength by use of the connecting strength value providedfrom the combining section and providing the extracted feature points tothe first and second camera motion calculating means. The edgeconnecting strength E is calculated by the combining section inaccordance with:

E=W _(G) G+W _(Cl) Cl+W _(S) S

[0018] wherein, W_(G) is a weighted value of a correlation matching,W_(Cl) is a weighted value of a location error matching, W_(S) is aweighted value of an edge strength matching, G is an edge strengthdifference between the feature point locations in the previous andcurrent images provided from the feature point extracting means, Cl is adistance difference between the current feature point location and thefeature point estimated location in the previous image provided from theoptical flow location estimating means, and S is a SUSAN edge strengthdifference between the feature points of the previous and current imagesprovided from the feature point extracting means.

[0019] The first camera motion calculating means comprises a pseudoinverse matrix calculating section for calculating a constant of acamera projection formula by use of the optical flow provided from theoptical flow calculating means; and a camera motion calculating sectionfor dividing the camera motion by use of the constant of the cameraprojection formula provided from the pseudo inverse matrix calculatingsection, and providing the resultant to the second camera motioncalculating means.

[0020] The second camera motion calculating means comprises an opticalflow direction and magnitude calculating section for calculating adirection and magnitude of the optical flow by use of a value of thecamera motion calculated by the first camera motion calculating means;an incorrect optical flow eliminating section for eliminating anincorrect optical flow among the optical flows provided from the opticalflow calculating means depending upon the direction and magnitude of theoptical flow calculated by an optical flow direction and magnitudecalculating section; and a camera motion calculating section forcalculating the camera motion by use of the optical flow which is noteliminated by the optical flow eliminating section, and providing thecamera motion to the weighted value calculating means and the opticalflow location estimating means, respectively. The direction andmagnitude of the optical flow are calculated by an optical flowdirection and magnitude calculating section in accordance with:$\begin{pmatrix}x_{Ave} \\y_{Ave}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- T_{x,y}}/Z_{t}} + {T_{z,t}{x_{t}/Z_{t}}} + {\Omega_{z,t}y_{t}}} \\{{{- T_{y,{t + 1}}}/Z_{t}} + {T_{z,t}{y_{t}/Z_{t}}} - {\Omega_{z,t}x_{t}}}\end{pmatrix}}}}$

[0021] wherein, Δt is a sampling time, T_(x), T_(y) and T_(z) are x, yand z-axis translation of the camera, respectively, and Ω_(z) is z-axisrotation of the camera, x_(t) and y_(t) are locations in the currentframe image, and x_(Ave) and y_(Ave) are standard locations. If adirection θ of a standard optical flow and a distance Δ between x_(Ave)and y_(Ave) are outside a predetermined range, the optical floweliminating section determines the calculated optical flow as amismatched optical flow to eliminate the incorrect optical flow.

[0022] The optical flow estimating means estimates the camera motion,depending upon the optical flow, in which the incorrect optical flow iseliminated, and the camera motion provided from the second camera motioncalculating means, in accordance with: $\begin{pmatrix}{\hat{x}}_{t + 1} \\{\hat{y}}_{t + 1}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- {\hat{T}}_{x,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{x_{t}/Z_{t}}} + {{\hat{\Omega}}_{z,{t + 1}}y_{t}}} \\{{{- {\hat{T}}_{y,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{y_{t}/Z_{t}}} - {{\hat{\Omega}}_{z,{t + 1}}x_{t}}}\end{pmatrix}}}}$

[0023] wherein, {circumflex over (T)}_(x,t+1), {circumflex over(T)}_(y,t+1), {circumflex over (T)}_(x,t+1), {circumflex over(Ω)}_(x,t+1) are camera motion estimated.

[0024] The weighted value calculating means comprises a correlationmatching weighted value calculating section for calculating acorrelation matching weighted value by use of the camera motion in whichthe incorrect optical flow is eliminated by the second camera motioncalculating means, and a weighted value normalized calculating sectionfor executing smoothing of a predetermined location estimated constantweighted value, an edge strength constant weighted value, and thecalculated correlation matching weighted value, and providing each ofthe smoothed values to the optical flow calculating means. Thecorrelation matching weighted value is calculated by the correlationmatching weighted value calculating section in accordance with:

W=W _(G,Inti) +K _(T) _(z) |T _(z) |+K _(Ω) _(z) |Ω_(z)|

[0025] wherein, W_(G,Init) is an initial set value of the correlationmatching weighted value.

[0026] The smoothing of the predetermined location estimated constantweighted value, the edge strength constant weighted value, and thecalculated correlation matching weighted value are executed by theweighted value normalized calculating section in accordance with:$W_{G} = \frac{W_{1}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}$$W_{Cl} = \frac{W_{{Cl},{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}$$W_{S} = \frac{W_{S,{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}$

[0027] wherein, W_(Cl,Init) and W_(S,Init) are the location estimatedweighted value and the edge strength constant weighted value,respectively.

[0028] It is to be understood that both the foregoing generaldescription and the following detailed description of the presentinvention are exemplary and explanatory and are intended to providefurther explanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are included to provide afurther understanding of the invention and are incorporated in andconstitute a part of this application, illustrate embodiment(s) of theinvention and together with the description serve to explain theprinciple of the invention. In the drawings:

[0030]FIG. 1 is a block diagram of an apparatus and method forcalculating an optical flow and a camera motion using a correlationmatching and a system model in moving images according to one preferredembodiment of the present invention;

[0031]FIG. 2 is a block diagram of the feature point extracting sectionin FIG. 1;

[0032]FIG. 3 is a block diagram of an optical flow calculating sectionin FIG. 1;

[0033]FIG. 4 is a block diagram of a first camera motion calculatingsection in FIG. 1;

[0034]FIG. 5 is a block diagram of the second camera motion calculatingsection 130 in FIG. 1;

[0035]FIG. 6 is a block diagram of an optical flow location estimatingsection in FIG. 1;

[0036]FIG. 7 is a block diagram of a weighted value calculating sectionin FIG. 1;

[0037]FIG. 8 is a view explaining incorrect optical evaluation in anoptical flow eliminating section in FIG. 5; and

[0038]FIG. 9 is a view showing an example tested by installing a camerain a coil spring system, in which it is a system model to apply alocation estimating method to an optical flow estimating section in FIG.1.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] An apparatus and method for estimating an optical flow and acamera motion using a correlation matching and a system model in movingimages according to one preferred embodiment of the present inventionwill now be explained with reference to the accompanying drawings.

[0040]FIG. 1 is a block diagram of an apparatus and method forcalculating an optical flow and a camera motion using a correlationmatching and a system model in moving images according to one preferredembodiment of the present invention. The apparatus of the presentinvention comprises an feature point extracting section 100 forextracting a feature point of a sequence of images (previous image andcurrent image) input from an image input section, an optical flowcalculating section 110 for calculating an optical flow by use of thefeature point extracted by the feature point extracting section, a firstcamera motion calculating section 120 for calculating a camera motion byuse of the optical flow calculated by the optical calculating section110, a second camera motion calculating section 130 for eliminating anincorrect optical flow in the optical flow calculated by the opticalflow calculating section 110 and the camera motion informationcalculated by the first camera motion calculating section 120 andcalculating the camera motion, an optical flow location estimatingsection 140 for estimating a location of the optical flow according tothe camera motion calculated by the second camera motion calculatingsection 130, and a weighted value calculating section 150 forcalculating a weighted value according to the camera motion calculatedby the second camera motion calculating section 130.

[0041] The detailed construction and operation of each section will nowbe explained with reference to the accompanying drawings.

[0042]FIG. 2 is a block diagram of the feature point extracting sectionin FIG. 1. As shown in FIG. 2, the feature point extracting section 100uses a SUSAN edge driving portion 101 and a local max portion 102 toextract a certain number of corner points, i.e., feature points, whichcan be obviously discriminated when being visually seen. The SUSAN edgedriving potion 101 finds the corner points in the edge image by use offollowing equation 1: $\begin{matrix}{S = {\left\{ \sum_{e^{- {(\frac{{I{({x,y})}} - {I{({{x + {dx}},{y + {dy}}})}}}{T})}^{6}}} \right\}/G}} & {{Equation}\quad 1}\end{matrix}$

[0043] wherein, I is a gray value, T is a threshold value for adifference between the gray values, and G is an edge strength differencebetween the feature point locations of previous and current image.

[0044] In the equation 1, if T is large, only a point in where a grayvalue variation is large is extracted as the feature point, while if Tis small, a point is where a gray value variation is small is alsoextracted as the feature point. A criterion of selecting the featurepoint is depended upon the value G. In order to obtain a certain numberof feature points, after a portion of an exponential function iscalculated while the value T is constant, the value G is got to bevaried.

[0045] In order to find the corner point in the edge image obtainedthrough the SUSAN edge driving portion 101, the local max portion 102selects the largest value in a set mask size area to extract the cornerpoint, i.e., the feature point.

[0046] If the number of the corner points calculated in the previousframe is smaller than a set range, the value G has to be reduced. If thenumber of feature numbers does not belong to the set range even thoughthe value is reduced, the value T set in a look up table is altered andsimultaneously the mask size is reduced in the local max portion 102,thereby assisting in extracting much feature points.

[0047] The location of the extracted feature point and the edge strengthof the location are provided to the optical flow calculating section110.

[0048]FIG. 3 is a block diagram of the optical flow calculating sectionin FIG. 1.

[0049] As shown in FIG. 3, a combining section 111 of the optical flowcalculating section 110 calculates an edge connecting strength inaccordance with the following equation 2:

E=W _(G) G+W _(Cl) Cl+W _(S) S  Equation 2

[0050] In other words, in the equation 2, W_(G), W_(Cl) and W_(S) areweighted values of a correlation matching, a location error matching andan edge strength matching, respectively. G is an edge strengthdifference between the feature point locations in the previous andcurrent images provided from the feature point extracting section 100.In addition, Cl is a distance difference between the current featurepoint location and a feature point estimated location in the previousimage provided from the optical flow location estimating section 140. Sis a SUSAN edge strength difference between the feature points of theprevious and current images provided from the feature point extractingsection 100.

[0051] The combining section 111 calculates the connecting strength E ofthe edge by use of the information of the distance difference betweenthe current feature point location and the feature point estimatedlocation in the previous image provided from the optical flow estimatingsection 140, the weighted values of the correlation, locationincorrectness, and edge strength matching provided from the featurepoint extracting section 100, and the edge strength difference betweenthe feature points of the previous and current images provided from thefeature point extracting section 100, and the calculated connectingstrength is supplied to the matched feature point extracting section112.

[0052] Accordingly, the matched feature point extracting section 112extracts the feature points of the largest connecting strength by use ofthe connecting strength value E provided from the combining section 111.The extracted feature points of the largest connecting strength areprovided to the first camera motion calculating section 120 and thesecond camera motion calculating section 130, respectively.

[0053] The first camera motion calculating section 120 substitutes theoptical flow received from the optical flow calculation section 110 in aformula induced by a system model and a camera projection to calculatesthe camera motion in accordance with a typical least square method.

[0054] A signal of the camera motion calculated thus is provided to thesecond camera motion calculating section 130. Herein, the first cameramotion calculating section 120 will now be explained with reference toFIG. 4.

[0055]FIG. 4 is a block diagram of the first camera motion calculatingsection in FIG. 1.

[0056] As shown in FIG. 4, a pseudo inverse matrix calculating sectionof the first camera motion calculating section 120 calculates a constantof a camera projection formula by use of the optical flow provided fromthe optical flow calculating section 110, and provides the constantvalue of the camera projection formula to the camera motion calculatingsection 122.

[0057] The camera motion calculating section 122 divides the cameramotion by use of the constant of the camera projection formula providedfrom the pseudo inverse matrix calculating section 121, and provides theresultant to the second camera motion calculating section 130.

[0058]FIG. 5 is a block diagram of the second camera motion calculatingsection 130 in FIG. 1.

[0059] As shown in FIG. 5, the second camera motion calculating section130 eliminates an incorrect optical flow among the optical flowsprovided from the optical flow calculating section 110 on the basis ofthe camera motion provided from the first camera motion calculatingsection 120. An optical flow direction and magnitude calculating section131 substitutes a value of the camera motion in the following equation 3combined with the camera projection formula and the camera motionformula to calculate the direction and magnitude of the optical flow:$\begin{matrix}{\begin{pmatrix}x_{Ave} \\y_{Ave}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- T_{x,y}}/Z_{t}} + {T_{z,t}{x_{t}/Z_{t}}} + {\Omega_{z,t}y_{t}}} \\{{{- T_{y,{t + 1}}}/Z_{t}} + {T_{z,t}{y_{t}/Z_{t}}} - {\Omega_{z,t}x_{t}}}\end{pmatrix}}}}} & {{Equation}\quad 3}\end{matrix}$

[0060] wherein, Δt is a sampling time, T_(x), T_(y) and T_(z) are x, yand z-axis translation of the camera, respectively, and Ω_(z) is z-axisrotation of the camera. X_(t) and y_(t) are locations in the currentframe image, and x_(Ave) and y_(Ave) are standard locations. If adirection θ of the standard optical flow and a distance A betweenx_(Ave) and y_(Ave) are outside the set range, as shown in FIG. 8, anincorrect optical flow eliminating section 132 determines the calculatedoptical flow as a mismatched optical flow to eliminate the incorrectoptical flow, and provides the residual optical flows to the cameramotion calculating section 133.

[0061] Specifically, after the incorrect optical flow among the opticalflows calculated by the optical flow direction and magnitude calculatingsection 131 in accordance with the equation 3 is eliminated by theincorrect optical flow eliminating section 132, the camera motioncalculating section 133 again calculates the camera motion by use of theresidual optical flow.

[0062] The calculated camera motion is provided to the optical flowlocation estimating section 140 and the weighted value calculatingsection 150, respectively.

[0063]FIG. 6 is a block diagram of the optical flow location estimatingsection in FIG. 1.

[0064] As shown in FIG. 6, the optical flow estimating section 140estimates the location in accordance with the following equation 4combined with the system model in which the camera is installed and thesystem camera projection, based on the optical flow and the cameramotion provided from the camera motion calculating section 130.$\begin{matrix}{\begin{pmatrix}{\hat{x}}_{t + 1} \\{\hat{y}}_{t + 1}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- {\hat{T}}_{x,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{x_{t}/Z_{t}}} + {{\hat{\Omega}}_{z,{t + 1}}y_{t}}} \\{{{- {\hat{T}}_{y,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{y_{t}/Z_{t}}} - {{\hat{\Omega}}_{z,{t + 1}}x_{t}}}\end{pmatrix}}}}} & {{Equation}\quad 4}\end{matrix}$

[0065] wherein, {circumflex over (T)}_(x,t+1), {circumflex over(T)}_(y,t+1), {circumflex over (T)}_(x,t+1), {circumflex over(Ω)}_(x,t+1) are camera motion estimated, and can be obtained from thesystem model in which the camera is installed. They are similar to asystem of FIG. 9 (a camera motion estimating calculating section 141using an equation 5 and a system model). In other words, FIG. 9 is aview showing an example tested by installing the camera in a coil springsystem, in which it is a system model to apply a location estimatingmethod to the optical flow estimating section 140.

{circumflex over (T)} _(x,t+1) =T _(x,t) +k ₁ {dot over (T)} _(x,t)

{circumflex over (T)} _(y,t+1) =T _(y,t) +k ₁ {dot over (T)} _(y,t)

{circumflex over (T)} _(z,t+1)=(mg+mT _(z,t) /Δt−k _(s) Z _(t))/(m/Δt+k_(s) Δt)

{circumflex over (Ω)}_(x,t+1)=(I _(z)Ω_(z,t) −k _(b)Θ_(z,t))/(I _(z)+Δtk _(b))

Θ_(z,t+1)=Θ_(z,t) +ΔtΩ _(z,t+1)  Equation 5

[0066] wherein, m is a weight of system (camera), g is an accelerationof gravity, K_(s) is a constant of a coil spring, k_(b) is a constant ofa rotational coil spring, I_(z) is a moment of secondary inertia, andθ_(z) is a torsion angle of the coil spring. In the equation 4,{circumflex over (x)}_(t+1), ŷ_(t+1) are estimated locations, and areprovided to the optical flow calculating section 110.

[0067]FIG. 7 is a block diagram of the weighted value calculatingsection in FIG. 1.

[0068] As shown in FIG. 7, the weighted value calculating section 150calculates a correlation matching weighted value by use of the cameramotion provided from the second camera motion calculating section 130after eliminating the incorrect optical flow, and provides apredetermined location estimated constant weighted value and an edgestrength constant weighted value to the optical flow calculating section110.

[0069] Since the reliability of the correlation matching value isreduced when an image rotating zoon phenomenon is happened, acorrelation matching weighted value calculating section 151 decreasesthe correlation matching weighted value as the following equation 6 whenoccurring the rotation and translation of an optical axis, i.e., az-axis, of the camera.

W=W _(G,Inti) +K _(T) _(z) |T _(z) |+K _(Ω) _(z) |Ω_(z)|  Equation 6

[0070] wherein, W_(G,Init) is an initial set value of the correlationmatching weighted value. A weighted value normalized calculating section152 shown in FIG. 7 executes the smoothing of the predetermined locationestimated constant weighted value, the edge strength constant weightedvalue, and the correlation matching variable weighted value calculatedby the correlation matching weighted value calculating section 151 inaccordance with the following equation 7, and provides the smoothedlocation estimated constant weighted value, the smoothed edge strengthweighted value, and the smoothed correlation matching weighted value tothe optical flow calculating section 110. $\begin{matrix}{{W_{G} = \frac{W_{1}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}}{W_{Cl} = \frac{W_{{Cl},{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}}{W_{S} = \frac{W_{S,{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}}} & {{Equation}\quad 7}\end{matrix}$

[0071] wherein, W_(Cl,Init) and W_(S,Init) are the location estimatedweighted value and the edge strength constant weighted value,respectively.

[0072] The method of the present invention may be recorded in arecording medium (for example, CD-ROM, RAM, ROM, floppy disc, hard disc,magneto-optical disc or the like) which is realized by a program to bereadable by a computer.

[0073] With the construction and process described above, according tothe apparatus and method for estimating an optical flow and a cameramotion using a correlation matching and a system model in moving imagesof the present invention the system motion in which the CCD camera isinstalled may be calculated by employing a method of precisely andrapidly the optical flow. That is, it can be applied to the motionextraction of an unmanned camera, an autonomic walking robot, afree-falling missile, an MPEG region or the like, in which quickcalculation is needed.

[0074] The forgoing embodiment is merely exemplary and is not to beconstrued as limiting the present invention. The present teachings canbe readily applied to other types of apparatuses. The description of thepresent invention is intended to be illustrative, and not to limit thescope of the claims. Many alternatives, modifications, and variationswill be apparent to those skilled in the art.

What is claimed is:
 1. An apparatus for calculating an optical flow anda camera motion using a correlation matching and a system model inmoving images, the apparatus comprising: feature point extracting meansfor extracting a feature point of a sequence of input images (previousimage and current image); optical flow calculating means for calculatingan optical flow by use of the feature point extracted by the featurepoint extracting means; first camera motion calculating means forcalculating a camera motion by use of the optical flow calculated by theoptical calculating means; second camera motion calculating means foreliminating an incorrect optical flow among the optical flows calculatedby the optical flow calculating means and again calculating aninformation of the camera motion; optical flow location estimating meansfor estimating a location of the optical flow by estimating aninformation of a distance difference between a feature point estimatedlocation and a current feature point location in the previous imageaccording to the camera motion calculated by the second camera motioncalculating means; and weighted value calculating means for calculatinga weighted value according to the camera motion calculated by the secondcamera motion calculating means and providing the weighted value to theoptical flow calculating means.
 2. The apparatus for calculating theoptical flow and camera motion as claimed in claim 1, wherein thefeature point extracting means comprises: a SUSAN edge driving portionfor extracting an edge image from the sequence of the input images; anda local max portion for selecting a largest value in a mask size regionset in the edge image extracted from the SUSAN edge extracting portionto extract a certain number of feature points.
 3. The apparatus forcalculating the optical flow and camera motion as claimed in claim 2,wherein the feature points are extracted by the local max portion inaccordance with:$S = {\left\{ \sum_{e^{- {(\frac{{I{({x,y})}} - {I{({{x + {dx}},{y + {dy}}})}}^{6}}{T})}}} \right\}/G}$

wherein, I is a gray value, T is a threshold value for a differencebetween the gray values, and G is an edge strength difference betweenfeature point locations of the previous and current images.
 4. Theapparatus for calculating the optical flow and camera motion as claimedin claim 1, wherein the optical flow calculating means comprises: acombining section for calculating a connecting strength of an edge byuse of the information of the distance difference between the currentfeature point location and the feature point estimated location in theprevious image provided from the optical flow estimating means, theweighted values of the correlation, location incorrectness, and edgestrength matching provided from the feature point extracting means, andthe edge strength difference between the feature points of the previousand current images provided from the feature point extracting means; anda feature point extracting section for extracting the feature points ofthe largest connecting strength by use of the connecting strength valueprovided from the combining section and providing the extracted featurepoints to the first and second camera motion calculating means.
 5. Theapparatus for calculating the optical flow and camera motion as claimedin claim 4, wherein the edge connecting strength E is calculated by thecombining section in accordance with: E=W _(G) G+W _(Cl) Cl+W _(S) Swherein, W_(G) is a weighted value of a correlation matching, W_(Cl) isa weighted value of a location error matching, W_(S) is a weighted valueof an edge strength matching, G is an edge strength difference betweenthe feature point locations in the previous and current images providedfrom the feature point extracting means, Cl is a distance differencebetween the current feature point location and the feature pointestimated location in the previous image provided from the optical flowlocation estimating means, and S is a SUSAN edge strength differencebetween the feature points of the previous and current images providedfrom the feature point extracting means.
 6. The apparatus forcalculating the optical flow and camera motion as claimed in claim 1,wherein the first camera motion calculating means comprises: a pseudoinverse matrix calculating section for calculating a constant of acamera projection formula by use of the optical flow provided from theoptical flow calculating means; and a camera motion calculating sectionfor dividing the camera motion by use of the constant of the cameraprojection formula provided from the pseudo inverse matrix calculatingsection, and providing the resultant to the second camera motioncalculating means.
 7. The apparatus for calculating the optical flow andcamera motion as claimed in claim 1, wherein the second camera motioncalculating means comprises: an optical flow direction and magnitudecalculating section for calculating a direction and magnitude of theoptical flow by use of a value of the camera motion calculated by thefirst camera motion calculating means; an incorrect optical floweliminating section for eliminating an incorrect optical flow among theoptical flows provided from the optical flow calculating means dependingupon the direction and magnitude of the optical flow calculated by anoptical flow direction and magnitude calculating section; and a cameramotion calculating section for calculating the camera motion by use ofthe optical flow which is not eliminated by the optical flow eliminatingsection, and providing the camera motion to the weighted valuecalculating means and the optical flow location estimating means,respectively.
 8. The apparatus for calculating the optical flow andcamera motion as claimed in claim 7, wherein the direction and magnitudeof the optical flow are calculated by an optical flow direction andmagnitude calculating section in accordance with: $\begin{pmatrix}x_{Ave} \\y_{Ave}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- T_{x,y}}/Z_{t}} + {T_{z,t}{x_{t}/Z_{t}}} + {\Omega_{z,t}y_{t}}} \\{{{- T_{y,{t + 1}}}/Z_{t}} + {T_{z,t}{y_{t}/Z_{t}}} - {\Omega_{z,t}x_{t}}}\end{pmatrix}}}}$

wherein, Δt is a sampling time, T_(x), T_(y) and T_(z) are x, y andz-axis translation of the camera, respectively, and Ω_(z) is z-axisrotation of the camera, x_(t) and y_(t) are locations in the currentframe image, and x_(Ave) and y_(Ave) are standard locations.
 9. Theapparatus for calculating the optical flow and camera motion as claimedin claim 8, wherein if a direction θ of a standard optical flow and adistance Δ between x_(Ave) and y_(Ave) are outside a predeterminedrange, the optical flow eliminating section determines the calculatedoptical flow as a mismatched optical flow to eliminate the incorrectoptical flow.
 10. The apparatus for calculating the optical flow andcamera motion as claimed in claim 1, wherein the optical flow estimatingmeans estimates the camera motion, depending upon the optical flow, inwhich the incorrect optical flow is eliminated, and the camera motionprovided from the second camera motion calculating means, in accordancewith: $\begin{pmatrix}{\hat{x}}_{t + 1} \\{\hat{y}}_{t + 1}\end{pmatrix} = {\begin{pmatrix}x_{t} \\y_{t}\end{pmatrix} + {\Delta \quad {t\begin{pmatrix}{{{- {\hat{T}}_{x,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{x_{t}/Z_{t}}} + {{\hat{\Omega}}_{z,{t + 1}}y_{t}}} \\{{{- {\hat{T}}_{y,{t + 1}}}/Z_{t}} + {{\hat{T}}_{z,{t + 1}}{y_{t}/Z_{t}}} - {{\hat{\Omega}}_{z,{t + 1}}x_{t}}}\end{pmatrix}}}}$

wherein, {circumflex over (T)}_(x,t+1), {circumflex over (T)}_(y,t+1),{circumflex over (T)}_(x,t+1), {circumflex over (Ω)}_(x,t+1) are cameramotion estimated.
 11. The apparatus for calculating the optical flow andcamera motion as claimed in claim 1, wherein the weighted valuecalculating means comprises: a correlation matching weighted valuecalculating section for calculating a correlation matching weightedvalue by use of the camera motion in which the incorrect optical flow iseliminated by the second camera motion calculating means; and a weightedvalue normalized calculating section for executing smoothing of apredetermined location estimated constant weighted value, an edgestrength constant weighted value, and the calculated correlationmatching weighted value, and providing each of the smoothed values tothe optical flow calculating means.
 12. The apparatus for calculatingthe optical flow and camera motion as claimed in claim 11, wherein thecorrelation matching weighted value is calculated by the correlationmatching weighted value calculating section in accordance with: W=W_(G,Inti) +K _(T) _(z) |T _(z) |+K _(Ω) _(z) |Ω_(z)|wherein, W_(G,Init)is an initial set value of the correlation matching weighted value. 13.The apparatus for calculating the optical flow and camera motion asclaimed in claim 11, wherein the smoothing of the predetermined locationestimated constant weighted value, the edge strength constant weightedvalue, and the calculated correlation matching weighted value areexecuted by the weighted value normalized calculating section inaccordance with: $\begin{matrix}{W_{G} = \frac{W_{1}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}} \\{W_{Cl} = \frac{W_{{Cl},{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}} \\{W_{S} = \frac{W_{S,{Init}}}{W_{1} + W_{{Cl},{Init}} + W_{S,{Init}}}}\end{matrix}$

wherein, W_(Cl,Init) and W_(S,Init) are the location estimated weightedvalue and the edge strength constant weighted value, respectively.